1. Technical Field
The present disclosure relates generally to an apparatus and method for providing augmented reality-based realistic, experience, and more particularly to an apparatus and method that accurately provide an augmented reality result, intended by a content creator, to a user who receives service when an augmented reality-based realistic experience content service is implemented using a large-sized screen, mirror-type display device.
2. Description of the Related Art
With the commercialization and popularization of RGBD sensors, such as Kinect, many realistic experience content services (see FIGS. 1 to 7), such as virtual dressing or virtual clothes fitting, directly targeted for users have appeared in connection with an augmented reality technique based on the recognition of known patterns, such as a book, a desk having a predetermined pattern, etc., which is used in existing e-learning, etc.
Such realistic experience content (1 of FIG. 1, and 2 of FIG. 2) provides augmented reality experience using a method of overlaying an object, having provided augmented reality intended by a realistic experience content creator, on a user image of a user moving in real time captured via various video image sensors. That is, a virtual mirror (3 of FIG. 3) is constructed using a video image sensor and a large-sized screen display panel, and an augmented reality object (4 of FIG. 4) is overlaid on a user region within an image of seeming to view a mirror, thereby providing the realistic experience of seeming to wear clothes.
An augmented reality technique using a virtual mirror (see FIGS. 1 and 2) is widely adopted due to the ease of implementation from the viewpoint of the configuration of a system. Although this technique provides the ease of implementation, most systems have delay time required for data processing within a system. Kinect has a delay time of about 100 msec. This delay time is inevitable when operations are required for a sensor and image processing in an augmented reality system. In order to overcome this, a method of preventing a user from becoming aware of the delay of an augmented reality result attributable to a system delay through the synchronization between the delay time of each sensor and image processing time may be used. Since the virtual mirror-based augmented reality enables the processing, many augmented reality-based realistic experience services use this method.
However, the virtual mirror-based method is disadvantageous in that image quality is inevitably degraded or a result of the virtual mirror is not viewed as realistic due to sensor characteristics because the virtual mirror-based method uses an image emitted from a video image sensor.
Recent approaches attempt mirror-based augmented reality realistic experience service using an actual mirror (see FIG. 5).
A mirror-based method using an actual mirror constructs a system in such a way as to combine a mirror 10 having two optical characteristics, as shown in FIG. 8, with a display panel 12, and provides service. In this case, the mirror 10 has a first optical characteristic adapted to reflect light 14 entering a mirror from the outside in a direction in which a user 5 views the light and thus exhibit an actual mirror effect, like an actual mirror, and a second optical characteristic adapted to pass light or an image 16 entering from the display panel 12 and thus enable the user 5 to view the light or image 16.
As an example, the combination of the mirror 10 and the display panel 12 is configured such that based on the viewpoint of a user, a mirror 10 is attached at a front location and a display panel 12 is placed at a rear location. When the display panel 12 does not emit an image, the combination operates as a mirror due to the first optical characteristic. When the display panel 12 emits an image, light 16 emitted by the display panel 12 based on the second optical characteristic is stronger than light 14 based on the first optical characteristic, and an image from the panel preferentially reaches the eyes of the user. When a currently commercialized mirror is taken as an example, about 80% of input light is transferred to the user with respect to the first optical characteristic, and about 30 to 40% of input light is transferred to the user with respect to the second optical characteristic.
That is, in the mirror-based method, due to the configuration of a system, a user recognizes 120˜130% light, into which two rays of light are added, with his or her eyes. Although the proportion of light transferred to a user can be relatively increased by increasing the candela of a display panel based on the second optical characteristic, a problem regarding the occurrence of the combination of colors attributable to the first optical characteristic and the second optical characteristic cannot be overcome. Accordingly, as shown in FIG. 7, a limitation arises in that the sensation of color of an augmented reality entity that a realistic experience content creator desires to transfer cannot be accurately transferred to a user. As an example, referring to FIG. 7, it can be seen that the color of virtual clothes, i.e., an augmented reality entity, is not accurately transferred due to the color of clothes worn by a user, unlike in the example of FIG. 4 using the virtual mirror-based method.
This mirror-based method is also different from the virtual mirror-based method in that an augmented reality entity presented to a display panel is intended only for a single user, as shown in the examples of FIGS. 5 to 7. In the case of the virtual mirror-based method, a final image obtained by combining an image received from a video image sensor with an augmented reality entity is presented via a display panel, as shown in FIGS. 3 and 4, and thus users located at various front locations view the same resulting screen. That is, when a service in which a user virtually wears clothes is taken as an example, a peer can determine the fitness of the clothes while viewing the same result along with the user.
Referring to Korean Patent Application Publication No. 2013-0128202 entitled “Mirror Display-based Augmented Reality method and Apparatus applied to the Same,” in a mirror-based method, such as that of FIG. 9, due to the first characteristic of a mirror 10, an augmented reality entity is presented (i.e., transmits the mirror 10; see reference symbol 16) at a virtual camera viewpoint 18 equivalent to the location of the eyes of a user 5 who is experiencing actual realistic experience content using a computer graphics rendering technology or an image processing technology via a display panel 12, thereby providing his or her own appearance 14 reflected on the mirror 10 and virtual clothes fitting experience (see FIG. 7).
In this case, a peer 7 views light 20 in a different direction, other than the appearance of the user 5 reflected on the mirror 10 based on the first optical characteristic of the mirror 10. Furthermore, an image from the display panel 12 is directly presented (see reference symbol 16) due to the second optical characteristic, and thus the user 5 and the augmented reality entity 16 are presented at different locations on a screen, thereby exhibiting the limitation in which the virtual wearing appearance of the user 5 cannot be viewed, as shown in FIG. 5 or 6. In FIG. 9, reference symbol 22 denotes a virtual camera viewpoint equivalent to the location of the eyes of the peer 7.
Other preceding technologies include Korean Patent Application Publication No. 2014-0077820 entitled “Virtual Item Fitting Method using Human Body Model and Virtual Item Fitting Service Provision System” and Korean Patent Application Publication No. 2014-0052792 entitled “Apparatus and Method for restoring 3D Entire Body Shape using Depth Image and Color Image.”